Motion vector refinement for multi-reference prediction

ABSTRACT

The present invention relates to the performance of motion vector refinement in a search space for multi-reference inter-prediction. Two or more reference pictures are selected, one of those used for motion vector refinement. Based on an initial estimate of a motion vector to the reference picture for motion vector refinement, a search space in this reference image is constructed. Using template matching, the first motion vector is refined. The second motion vector to another reference picture is calculated using its initial estimate, the initial estimate of the first motion vector and the refined first motion vector.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S. patent application Ser. No.16/728,747, filed on Dec. 27, 2019, which is a continuation ofInternational Application No. PCT/EP2018/058203, filed on Mar. 29, 2018.The International Application claims priority to International PatentApplication No. PCT/EP2017/066342, filed on Jun. 30, 2017. All of theafore-mentioned patent applications are hereby incorporated by referencein their entireties.

BACKGROUND

Current hybrid video codecs employ predictive coding. A picture of avideo sequence is subdivided into blocks of pixels and these blocks arethen coded. Instead of coding a block pixel by pixel, the entire blockis predicted using already encoded pixels in the spatial or temporalproximity of the block. The encoder further processes only thedifferences between the block and its prediction. The further processingtypically includes a transformation of the block pixels intocoefficients in a transformation domain. The coefficients may then befurther compressed by means of quantization and further compacted byentropy coding to form a bitstream. The bitstream further includes anysignaling information which enables the decoder to decode the encodedvideo. For instance, the signaling may include settings concerning theencoder settings such as size of the input picture, frame rate,quantization step indication, prediction applied to the blocks of thepictures, or the like.

Temporal prediction exploits temporal correlation between pictures, alsoreferred to as frames, of a video. The temporal prediction is alsocalled inter-prediction, as it is a prediction using the dependenciesbetween (inter) different video frames. Accordingly, a block beingencoded, also referred to as a current block, is predicted from (one or)more previously encoded picture(s) referred to as a reference picture.The reference picture is not necessarily a picture preceding the currentpicture in which the current block is located in the displaying order ofthe video sequence. The encoder may encode the pictures in a codingorder different from the displaying order. As a prediction of thecurrent block, a co-located block in a reference picture may bedetermined. The co-located block is a block which is located in thereference picture on the same position as the current block in thecurrent picture. Such prediction is accurate for motionless pictureregions, i.e. picture regions without movement from one picture toanother.

In order to obtain a predictor which takes into account the movement,i.e. a motion compensated predictor, motion estimation is typicallyemployed when determining the prediction of the current block.Accordingly, the current block is predicted by a block in the referencepicture, which is located in a distance given by a motion vector fromthe position of the co-located block. In order to enable a decoder todetermine the same prediction of the current block, the motion vectormay be signaled in the bitstream. In order to further reduce thesignaling overhead caused by signaling the motion vector for each of theblocks, the motion vector itself may be estimated. The motion vectorestimation may be performed based on the motion vectors of theneighboring blocks in spatial and/or temporal domain.

The prediction of the current block may be computed using one referencepicture or by weighting predictions obtained from two or more referencepictures. The reference picture may be an adjacent picture, i.e. apicture immediately preceding and/or the picture immediately followingthe current picture in the display order since adjacent pictures aremost likely to be similar to the current picture. However, in general,the reference picture may be also any other picture preceding orfollowing the current picture in the displaying order and preceding thecurrent picture in the bitstream (decoding order). This may provideadvantages for instance in case of occlusions and/or non-linear movementin the video content. The reference picture may thus be also signaled inthe bitstream.

A special mode of the inter-prediction is a so-called bi-prediction inwhich two reference pictures are used in generating the prediction ofthe current block. In particular, two predictions determined in therespective two reference pictures are combined into a prediction signalof the current block. The bi-prediction may result in a more accurateprediction of the current block than the uni-prediction, i.e. predictiononly using a single reference picture. The more accurate predictionleads to smaller differences between the pixels of the current block andthe prediction (referred to also as “residuals”), which may be encodedmore efficiently, i.e. compressed to a shorter bitstream. In general,more than two reference pictures may be used to find respective morethan two reference blocks to predict the current block, i.e. amulti-reference inter prediction can be applied. The termmulti-reference prediction thus includes bi-prediction as well aspredictions using more than two reference pictures.

In order to provide more accurate motion estimation, the resolution ofthe reference picture may be enhanced by interpolating samples betweenpixels. Fractional pixel interpolation can be performed by weightedaveraging of the closest pixels. In case of half-pixel resolution, forinstance a bilinear interpolation is typically used. Other fractionalpixels are calculated as an average of the closest pixels weighted bythe inverse of the distance between the respective closest pixels to thepixel being predicted.

The motion vector estimation is a computationally complex task in whicha similarity is calculated between the current block and thecorresponding prediction blocks pointed to by candidate motion vectorsin the reference picture. In order to reduce the complexity, the numberof candidate motion vectors is usually reduced by limiting the candidatemotion vectors to a certain search space. The search space may be, forinstance, defined by a number and/or positions of pixels surrounding theposition in the reference picture corresponding to the position of thecurrent block in the current image. On the other hand, the candidatemotion vectors may be defined by a list of candidate motion vectorsformed by motion vectors of neighboring blocks.

Motion vectors are usually at least partially determined at the encoderside and signaled to the decoder within the coded bitstream. However,the motion vectors may also be derived at the decoder. In such case, thecurrent block is not available at the decoder and cannot be used forcalculating the similarity to the blocks to which the candidate motionvectors point in the reference picture. Therefore, instead of thecurrent block, a template is used which is constructed out of pixels ofalready decoded blocks. For instance, already decoded pixels adjacent tothe current block (in the current picture or in the reference picture)may be used. Such motion estimation provides an advantage of reducingthe signaling: the motion vector is derived in the same way at both theencoder and the decoder and thus, no signaling is needed. On the otherhand, the accuracy of such motion estimation may be lower.

In order to provide a tradeoff between the accuracy and signalingoverhead, the motion vector estimation may be divided into two steps:motion vector derivation and motion vector refinement. For instance, amotion vector derivation may include selection of a motion vector fromthe list of candidates. Such a selected motion vector may be furtherrefined for instance by a search within a search space. The search inthe search space is based on calculating cost function for eachcandidate motion vector, i.e. for each candidate position of block towhich the candidate motion vector points.

Document JVET-D0029: Decoder-Side Motion Vector Refinement Based onBilateral Template Matching, X. Chen, J. An, J. Zheng (The document canbe found at: http://phenix.it-sudparis.eu/jvet/site) shows motion vectorrefinement in which a first motion vector in integer pixel resolution isfound and further refined by a search with a half-pixel resolution in asearch space around the first motion vector.

SUMMARY

When multi-reference prediction is applied, motion vectors in aplurality of reference pictures have to be determined. Even if themotion vectors are signaled in the first stage so that the decoder doesnot need to perform any further search, the motion vector refinementstill requires an additional search among the motion vectors of acorresponding search space. This may be a complex task which requirescomputational resources as well as memory.

The present disclosure provides a technique in which a first motionvector in a first reference picture and a second motion vector in asecond reference picture are determined. Complexity can thus be reduced.At first, the first motion vector and the second motion vector areroughly estimated. Then, the first motion vector is refined byperforming a search in a search space given by the rough estimate of thefirst motion vector. The second motion vector is determined bycalculation based on its rough estimate as well as based on the refinedfirst motion vector. The first and second motion vectors may be appliedin inter-prediction of the current block in a current picture, used atthe decoder on the encoding and/or decoding side.

According to a first aspect, the invention relates to an apparatus fordetermining a first motion vector in a first reference picture and asecond motion vector in a second reference picture, the first and secondmotion vectors to be applied in inter-prediction of a picture block in acurrent picture, the apparatus comprises a motion vector refinement unitand a motion vector calculation unit. The motion vector refinement unitis configured to obtain an estimate of the first motion vector. A searchspace is specified based on the estimate of the first motion vector.Within the search space, the motion vector refinement unit performssearch to determine the first motion vector. The motion vectorcalculation unit obtains an estimate of the second motion vector. Basedon the estimate of the second motion vector and the first motion vector,the motion vector calculation unit calculates the second motion vector.

Thus, the motion vector determination can be performed with lesscomplexity while still maintaining accuracy given by the refinement ofthe first motion vector and based thereon estimating the refinementamount for the second motion vector.

In a possible implementation form of the apparatus according to thefirst aspect as such, the second motion vector is calculated by addingto the estimate of the second motion vector a function of a differencebetween the first motion vector and the estimate of the first motionvector. This function may include scaling and/or clipping. When thescaling parameter is determined, its value may depend on the ratiobetween the respective distances of the first reference picture and thesecond reference picture to the current picture.

The calculation of the second motion vector as a function of therefinement performed on the first motion vector is a low-complexityestimation. Moreover, by further modifying it (e.g., by scalingdepending on the distance between the respective reference pictures),the estimation may become even more accurate.

The apparatus, advantageously further contains a reference pictureselection unit for obtaining reference pictures and selecting which ofthem is the first reference picture and the second reference picture.Following this selection it determines whether the first referencepicture or the second reference picture is to be used for motion vectorrefinement. A list of reference pictures associates an index to beincluded into bitstream with the position of a reference picture withrespect to the current picture. The reference picture selection unit isconfigured to select the first reference picture and the secondreference picture based on whether or not it is referred to in thebitstream by an index in the predefined list of reference pictures.

In other words, the reference picture selection unit is configured toselect either the first picture or the second picture based on whetherthe respective first or second picture is referred to in a bit streamalso including the coded picture block of the video by an index relatingto a predefined list of reference pictures among at least two lists ofreference pictures, wherein a list of reference pictures associates anindex with the position of a reference picture with respect to thecurrent picture.

If the two reference pictures are referred to in the bitstream by anindex in the same predefined list of reference pictures, the referencepicture selection unit may select the reference picture to be used formotion vector refinement as the picture with the highest position insaid list of reference pictures.

Alternatively, the reference picture to be used for motion vectorrefinement can be selected as the picture, which has the lowest temporallayer among the two pictures.

The reference picture to be used for motion vector refinement can beselected as the picture, which has the lowest base quantization value,or as the picture, which has the lowest distance to the current pictureas well.

As a further approach, the reference picture to be used for motionvector refinement can be selected such that the magnitude of theestimate of the motion vector pointing at the reference picture to beused for motion vector refinement is smaller than the magnitude of theestimate of the other motion vector.

The apparatus may further comprise a motion vector determination unitfor determining the estimate of the first motion vector and the estimateof the second motion vector. This determination is performed byselecting them from a set of motion vector candidates based onsimilarity of a template with a portion of the respective picturereferred to by the motion vector candidate.

A video encoder for encoding a plurality of pictures into a bitstreamcomprises an inter-prediction unit, a bitstream former and areconstruction unit. The inter-prediction unit includes the apparatusfor determining the first motion vector and the second motion vector aswell as a prediction unit. The prediction unit determines the predictionblock according to a portion of the first reference picture referred toby the first motion vector and a portion of the second reference picturereferred to by the second motion vector. The bitstream former includesinto the bitstream the estimate of the first motion vector and theestimate of the second motion vector. The reconstruction unitreconstructs the current block according to the prediction block andstores the reconstructed block into a memory.

A video decoder for decoding a plurality of pictures from a bitstreamcomprises an inter-prediction unit, a bitstream parser and areconstruction unit. The inter-prediction unit includes the apparatusfor determining the first motion vector and the second motion vector aswell as a prediction unit. The prediction unit determines the predictionblock according to a portion of the first reference picture referred toby the first motion vector and a portion of the second reference picturereferred to by the second motion vector. The bitstream parser obtainsfrom the bitstream an estimate of the first motion vector and anestimate of the second motion vector. The reconstruction unitreconstructs the current block, according to the prediction block.

The method comprises motion vector refinement and motion vectorcalculation. An estimate of the first motion vector is obtained. Asearch space is specified based on the estimate of the first motionvector. Within the search space, a search to determine the first motionvector is performed. An estimate of the second motion vector isobtained. Based on the estimate of the second motion vector and thefirst motion vector, the second motion vector is calculated. The secondmotion vector is calculated by adding to the estimate of the secondmotion vector a function of a difference between the first motion vectorand the estimate of the first motion vector. This function includesscaling and/or clipping. The value of the scaling parameter depends onthe ratio between the respective distances of the first referencepicture and the second reference picture to the current picture.

The method further comprises a reference picture selection for obtainingreference pictures and selecting which of them is the first referencepicture and the second reference picture. Following this selection itdetermines whether the first reference picture or the second referencepicture is to be used for motion vector refinement. A list of referencepictures associates an index to be included into bitstream with theposition of a reference picture with respect to the current picture. Thereference picture selection is performed to select the first referencepicture and the second reference picture based on whether or not it isreferred to in the bitstream by an index in the predefined list ofreference pictures. If the two reference pictures are referred to in thebitstream by an index in the same predefined list of reference pictures,the reference picture to be used for motion vector refinement isselected as the picture with the highest position in said list ofreference pictures. Alternatively, the reference picture to be used formotion vector refinement can be selected as the picture, which has thelowest temporal layer among the two pictures. The reference picture tobe used for motion vector refinement can be selected as the picture,which has the lowest base quantization value, or as the picture, whichhas the lowest distance to the current picture as well. As a furtherapproach, the reference picture to be used for motion vector refinementcan be selected such that the magnitude of the estimate of the motionvector pointing to the reference picture to be used for motion vectorrefinement is smaller than the magnitude of the estimate of the othermotion vector.

The method may further determine the estimate of the first motion vectorand the estimate of the second motion vector. This determination isperformed by selecting them from a set of motion vector candidates basedon similarity of a template with a portion of the respective picturereferred to by the motion vector candidate.

A video encoding method for encoding a plurality of pictures into abitstream comprises performing inter-prediction, bitstream forming andblock reconstruction. The inter-prediction includes determining thefirst motion vector and the second motion vector as well as a blockprediction. The prediction includes determining the prediction blockaccording to a portion of the first reference picture referred to by thefirst motion vector and a portion of the second reference picturereferred to by the second motion vector. The bitstream forming includesinto the bitstream the estimate of the first motion vector and theestimate of the second motion vector. The reconstruction includesreconstructing the current block according to the prediction block andstoring the reconstructed block into a memory.

A video decoding method for decoding a plurality of pictures from abitstream comprises performing inter-prediction, bitstream parsing andblock reconstruction. The inter-prediction includes determining thefirst motion vector and the second motion vector as well as a blockprediction. The prediction determines the prediction block according toa portion of the first reference picture referred to by the first motionvector and a portion of the second reference picture referred to by thesecond motion vector.

The bitstream parsing obtains from the bitstream an estimate of thefirst motion vector and an estimate of the second motion vector. Thereconstruction includes reconstructing the current block, according tothe prediction block.

The present invention may reduce the number of search candidates in theprocess of motion vector refinement without any impact on codingperformance while providing similar picture quality. This is achieved byperforming a search for the motion vector refinement only on onereference picture for a current block, while another motion vector foranother reference picture of the same current block is calculated basedon the refined motion vector.

BRIEF DESCRIPTION OF THE DRAWINGS

In the following exemplary embodiments are described in more detail withreference to the attached figures and drawings, in which:

FIG. 1 is a block diagram showing an exemplary structure of an encoderin which the motion vector derivation and refinement may be employed;

FIG. 2 is a block diagram showing an exemplary structure of a decoder inwhich the motion vector derivation and refinement may be employed;

FIG. 3 is a schematic drawing illustrating motion vector refinement of afirst motion vector;

FIG. 4 is a schematic drawing illustrating motion vector refinement ofthe second motion vector based on the refined first motion vector;

FIG. 5 is a block diagram illustrating a motion vector refinementapparatus;

FIG. 6 is a schematic drawing illustrating various configurations ofsearch spaces;

FIG. 7 is a schematic drawing illustrating a method for determiningmotion vectors according to an embodiment; and

FIG. 8 is a block diagram illustrating an inter-prediction unit fordetermining motion vectors and performing intra-prediction according toan embodiment.

DESCRIPTION OF THE EMBODIMENTS

The present disclosure relates to determination of motion vectors for amulti-reference prediction. It may be employed in motion estimationperformed during encoding and decoding of video. In the following,exemplary encoder and decoder which may implement the motion estimationemploying the search space construction of the present disclosure aredescribed in the following.

FIG. 1 shows an encoder 100 which comprises an input for receiving inputblocks of frames or pictures of a video stream and an output forgenerating an encoded video bitstream. The term “frame” in thisdisclosure is used as a synonym for picture. However, it is noted thatthe present disclosure is also applicable to fields in case interlacingis applied. In general, a picture includes m times n pixels. Thiscorresponds to image samples and may comprise one or more colorcomponents. For the sake of simplicity, the following description refersto pixels meaning samples of luminance. However, it is noted that themotion vector search of the invention can be applied to any colorcomponent including chrominance or components of a search space such asRGB or the like. On the other hand, it may be beneficial to only performmotion vector estimation for one component and to apply the determinedmotion vector to more (or all) components.

The input blocks to be coded do not necessarily have the same size. Onepicture may include blocks of different sizes and the block raster ofdifferent pictures may also differ.

In an explicative realization, the encoder 100 is configured to applyprediction, transformation, quantization, and entropy coding to thevideo stream. The transformation, quantization, and entropy coding arecarried out respectively by a transform unit 101, a quantization unit102 and an entropy encoding unit 103 so as to generate as an output theencoded video bitstream.

The video stream may include a plurality of frames, wherein each frameis divided into blocks of a certain size that are either intra or intercoded. The blocks of for example the first frame of the video stream areintra coded by means of an intra prediction unit 109. An intra frame iscoded using only the information within the same frame, so that it canbe independently decoded and it can provide an entry point in thebitstream for random access. Blocks of other frames of the video streammay be inter coded by means of an inter prediction unit 110: informationfrom previously coded frames (reference frames) is used to reduce thetemporal redundancy, so that each block of an inter-coded frame ispredicted from a block in a reference frame. A mode selection unit 108is configured to select whether a block of a frame is to be processed bythe intra prediction unit 109 or the inter prediction unit 110. Thisblock also controls the parameters of intra of inter prediction. Inorder to enable refreshing of the image information, intra-coded blockmay be provided within inter-coded frames. Moreover, intra-frames whichcontain only intra-coded blocks may be regularly inserted into the videosequence in order to provide entry points for decoding, i.e. pointswhere the decoder can start decoding without having information from thepreviously coded frames.

The intra prediction unit 109 is a block prediction unit. For performingspatial or temporal prediction, the coded blocks may be furtherprocessed by an inverse quantization unit 104, and an inverse transformunit 105. After reconstruction of the block a loop filtering unit 106 isapplied to further improve the quality of the decoded image. Thefiltered blocks then form the reference frames that are then stored in aframe buffer 107. Such decoding loop (decoder) at the encoder sideprovides the advantage of producing reference frames which are the sameas the reference pictures reconstructed at the decoder side.Accordingly, the encoder and decoder side operate in a correspondingmanner. The term “reconstruction” here refers to obtaining thereconstructed block by adding to the decoded residual block theprediction block.

The inter prediction unit 110 receives as an input a block of a currentframe or picture to be inter coded and one or several reference framesor pictures from the frame buffer 107. Motion estimation and motioncompensation are applied by the inter prediction unit 110. The motionestimation is used to obtain a motion vector and a reference frame basedon certain cost function. The motion compensation then describes acurrent block of the current frame in terms of the translation of areference block of the reference frame to the current frame, i.e. by amotion vector. The inter prediction unit 110 outputs the predictionblock for the current block, wherein said prediction block minimizes thecost function. For instance, the cost function may be a differencebetween the current block to be coded and its prediction block, i.e. thecost function minimizes the residual block. The minimization of theresidual block is based e.g. on calculating a sum of absolutedifferences (SAD) between all pixels (samples) of the current block andthe candidate block in the candidate reference picture. However, ingeneral, any other similarity metric may be employed, such as meansquare error (MSE) or structural similarity metric (SSIM).

However, cost-function may also be the number of bits necessary to codesuch inter-block and/or distortion resulting from such coding. Thus, therate-distortion optimization procedure may be used to decide on themotion vector selection and/or in general on the encoding parameterssuch as whether to use inter or intra prediction for a block and withwhich settings.

The intra prediction unit 109 receives as an input a block of a currentframe or picture to be intra coded and one or several reference samplesfrom an already reconstructed area of the current frame. The intraprediction then describes pixels of a current block of the current framein terms of a function of reference samples of the current frame. Theintra prediction unit 109 outputs a prediction block for the currentblock, wherein said prediction block advantageously minimizes thedifference between the current block to be coded and its predictionblock, i.e., it minimizes the residual block. The minimization of theresidual block can be based e.g. on a rate-distortion optimizationprocedure. In particular, the prediction block is obtained as adirectional interpolation of the reference samples. The direction may bedetermined by the rate-distortion optimization and/or by calculating asimilarity measure as mentioned above in connection withinter-prediction.

The difference between the current block and its prediction, i.e. theresidual block, is then transformed by the transform unit 101. Thetransform coefficients are quantized by the quantization unit 102 andentropy coded by the entropy encoding unit 103. The thus generatedencoded video bitstream comprises intra coded blocks and inter codedblocks and the corresponding signaling (such as the mode indication,indication of the motion vector, and/or intra-prediction direction). Thetransform unit 101 may apply a linear transformation such as a Fourieror Discrete Cosine Transformation (DFT/FFT or DCT). Such transformationinto the spatial frequency domain provides the advantage that theresulting coefficients have typically higher values in the lowerfrequencies. Thus, after an effective coefficient scanning (such aszig-zag), and quantization, the resulting sequence of values hastypically some larger values at the beginning and ends with a run ofzeros. This enables further efficient coding. Quantization unit 102performs the actual lossy compression by reducing the resolution of thecoefficient values. The entropy coding unit 103 then assigns tocoefficient values binary codewords to produce a bitstream. The entropycoding unit 103 also codes the signaling information (not shown in FIG.1 ).

FIG. 2 shows a video decoder 200. The video decoder 200 comprisesparticularly a reference picture buffer 207 and an intra prediction unit209, which is a block prediction unit. The reference picture buffer 207is configured to store at least one reference frame reconstructed fromthe encoded video bitstream, said reference frame being different from acurrent frame (currently decoded frame) of the encoded video bitstream.The intra prediction unit 209 is configured to generate a predictionblock, which is an estimate of the block to be decoded. The intraprediction unit 209 is configured to generate this prediction based onreference samples that are obtained from the reference picture buffer207.

The decoder 200 is configured to decode the encoded video bitstreamgenerated by the video encoder 100, and preferably both the decoder 200and the encoder 100 generate identical predictions for the respectiveblock to be encoded/decoded. The features of the reference picturebuffer 207 and the intra prediction unit 209 are similar to the featuresof the reference picture buffer 107 and the intra prediction unit 109 ofFIG. 1 .

The video decoder 200 comprises further units that are also present inthe video encoder 100 like e.g. an inverse quantization unit 204, aninverse transform unit 205, and a loop filtering unit 206, whichrespectively correspond to the inverse quantization unit 104, theinverse transform unit 105, and the loop filtering unit 106 of the videoencoder 100.

An entropy decoding unit 203 is configured to decode the receivedencoded video bitstream and to correspondingly obtain quantized residualtransform coefficients and signaling information. The quantized residualtransform coefficients are fed to the inverse quantization unit 204 andan inverse transform unit 205 to generate a residual block. The residualblock is added to a prediction block and the addition is fed to the loopfiltering unit 206 to obtain the decoded video. Frames of the decodedvideo can be stored in the reference picture buffer 207 and serve as areference frame for inter prediction.

Generally, the intra prediction units 109 and 209 of FIGS. 1 and 2 canuse reference samples from an already encoded area to generateprediction signals for blocks that need to be encoded or need to bedecoded.

The entropy decoding unit 203 receives as its input the encodedbitstream. In general, the bitstream is at first parsed, i.e. thesignaling parameters and the residuals are extracted from the bitstream.Typically, the syntax and semantic of the bitstream is defined by astandard so that the encoders and decoders may work in an interoperablemanner. As described in the above Background section, the encodedbitstream does not only include the prediction residuals. In case ofmotion compensated prediction, a motion vector indication is also codedin the bitstream and parsed therefrom at the decoder. The motion vectorindication may be given by means of a reference picture in which themotion vector is provided and by means of the motion vector coordinates.So far, coding the complete motion vectors was considered. However, alsoonly the difference between the current motion vector and the previousmotion vector in the bitstream may be encoded. This approach allowsexploiting the redundancy between motion vectors of neighboring blocks.

In order to efficiently code the reference picture, H.265 codec (ITU-T,H265, Series H: Audiovisual and multimedia systems: High Efficient VideoCoding) provides a list of reference pictures assigning to list indicesrespective reference frames. The reference frame is then signaled in thebitstream by including therein the corresponding assigned list index.Such list may be defined in the standard or signaled at the beginning ofthe video or a set of a number of frames. It is noted that in H.265there are two lists of reference pictures defined, called L0 and L1. Thereference picture is then signaled in the bitstream by indicating thelist (L0 or L1) and indicating an index in that list associated with thedesired reference picture. Providing two or more lists may haveadvantages for better compression. For instance, L0 may be used for bothuni-directionally inter-predicted slices and bi-directionallyinter-predicted slices while L1 may only be used for bi-directionallyinter-predicted slices. However, in general the present disclosure isnot limited to any content of the L0 and L1 lists.

The lists L0 and L1 may be defined in the standard and fixed. However,more flexibility in coding/decoding may be achieved by signaling them atthe beginning of the video sequence. Accordingly, the encoder mayconfigure the lists L0 and L1 with particular reference pictures orderedaccording to the index. The L0 and L1 lists may have the same fixedsize. There may be more than two lists in general. The motion vector maybe signaled directly by the coordinates in the reference picture.Alternatively, as also specified in H.265, a list of candidate motionvectors may be constructed and an index associated in the list with theparticular motion vector can be transmitted.

Motion vectors of the current block are usually correlated with themotion vectors of neighboring blocks in the current picture or in theearlier coded pictures. This is because neighboring blocks are likely tocorrespond to the same moving object with similar motion and the motionof the object is not likely to change abruptly over time. Consequently,using the motion vectors in neighboring blocks as predictors reduces thesize of the signaled motion vector difference. The MVPs are usuallyderived from already decoded motion vectors from spatial neighboringblocks or from temporally neighboring blocks in the co-located picture.In H.264/AVC, this is done by doing a component wise median of threespatially neighboring motion vectors. Using this approach, no signalingof the predictor is required. Temporal MVPs from a co-located pictureare only considered in the so called temporal direct mode of H.264/AVC.The H.264/AVC direct modes are also used to derive other motion datathan the motion vectors. Hence, they relate more to the block mergingconcept in HEVC. In HEVC, the approach of implicitly deriving the MVPwas replaced by a technique known as motion vector competition, whichexplicitly signals which MVP from a list of MVPs, is used for motionvector derivation. The variable coding quadtree block structure in HEVCcan result in one block having several neighboring blocks with motionvectors as potential MVP candidates. Taking the left neighbor as anexample, in the worst case a 64×64 luma prediction block could have 164×4 luma prediction blocks to the left when a 64×64 luma coding treeblock is not further split and the left one is split to the maximumdepth.

Advanced Motion Vector Prediction (AMVP) was introduced to modify motionvector competition to account for such a flexible block structure.During the development of HEVC, the initial AMVP design wassignificantly simplified to provide a good trade-off between codingefficiency and an implementation friendly design. The initial design ofAMVP included five MVPs from three different classes of predictors:three motion vectors from spatial neighbors, the median of the threespatial predictors and a scaled motion vector from a co-located,temporally neighboring block. Furthermore, the list of predictors wasmodified by reordering to place the most probable motion predictor inthe first position and by removing redundant candidates to assureminimal signaling overhead. The final design of the AMVP candidate listconstruction includes the following two MVP candidates: a) up to twospatial candidate MVPs that are derived from five spatial neighboringblocks; b) one temporal candidate MVPs derived from two temporal,co-located blocks when both spatial candidate MVPs are not available orthey are identical; and c) zero motion vectors when the spatial, thetemporal or both candidates are not available. Details on motion vectordetermination can be found in the book by V. Sze et al (Ed.), HighEfficiency Video Coding (HEVC): Algorithms and Architectures, Springer,2014, in particular in Chapter 5, incorporated herein by reference.

In order to further improve motion vector estimation without furtherincrease in signaling overhead, it may be beneficial to further refinethe motion vector derived at the encoder side and provided in thebitstream. The motion vector refinement may be performed at the decoderwithout assistance from the encoder. The encoder in its decoder loop mayemploy the same refinement to obtain corresponding reference pictures.The refinement is performed by determining a template, determining asearch space and finding in the search space a reference picture portionbest matching the template. The best matching portion positiondetermines the best motion vector which is then used to obtain thepredictor of the current block, i.e. the current block beingreconstructed.

FIG. 3 illustrates a search space (search region) 310 which includesinteger pixel positions (full dots) and fractional pixel positions(empty dots) of a reference picture. In this example, the fractionalpixel positions are half-pixel positions. As described above, thefractional pixel positions may be obtained from the integer (full-pixel)positions by interpolation such as bi-linear interpolation.

In a bi-prediction of current block, two prediction blocks obtainedusing the respective first motion vector of list L0 and the secondmotion vector of list L1, are combined to a single prediction signal,which can provide a better adaptation to the original signal thanuni-prediction, resulting in less residual information and possibly amore efficient compression. FIG. 3 further illustrates a current block320 from the current picture. Since at the decoder, the current block isnot available since it is being decoded, for the purpose of motionvector refinement, a template is used, which is an estimate of thecurrent block and which is constructed based on the already processed(i.e. coded at the encoder side and decoded at the decoder side) imageportions.

The template may be constructed, for example, based on samples belong tothe current picture which have already been decoded, i.e. which aredecoded before the current block. Additionally or alternatively, thesesamples may belong to any of the previously decoded reference pictures.As an example, the samples to be used in construction of the templatemay belong to the reference picture that is decoded before the currentpicture and precedes the current picture in display order.Alternatively, the samples may belong to the reference picture that isdecoded before the current picture and that follows the current picturein display order. Finally, the template can be constructed based on acombination of samples from two different reference pictures. As isapparent to the person skilled in the art, the template can be obtainedusing different methods such that the current block can be estimatedwith the constructed template.

First, an estimate of the first motion vector MV0 and an estimate of thesecond motion vector MV1 are received as input at the decoder 200 asillustrated in FIG. 3 . At the encoder side 100, the motion vectorestimates MV0 and MV1 may be obtained by block matching and/or by searchin a list of candidates (such as merge list) formed by motion vectors ofthe blocks neighboring to the current block (in the same picture or inadjacent pictures). The MV0 and MV1 are then advantageously signaled tothe decoder side within the bitstream. However, it is noted that ingeneral, also the first determination stage at the encoder could beperformed by template matching which would provide the advantage ofreducing signaling overhead.

At the decoder side 200, the motion vectors MV0 and MV1 areadvantageously obtained based on information in the bitstream. The MV0and MV1 are either directly signaled, or differentially signaled, and/oran index in the list of motion vector (merge list) is signaled. However,the present disclosure is not limited to signaling motion vectors in thebitstream. Rather, the motion vector may be determined by templatematching already in the first stage, correspondingly to the operation ofthe encoder. The template matching of the first stage (motion vectorderivation) may be performed based on a search space different from thesearch space of the second, refinement stage. In particular, therefinement may be performed on a search space with higher resolution(i.e. lower distance between the search positions).

An indication of the two reference pictures, to which respective MV0 andMV1 point, are provided to the decoder as well. The reference picturesare stored in the reference picture buffer at the encoder and decoderside as a result of previous processing, i.e. respective encoding anddecoding. One of these reference pictures is selected for motion vectorrefinement by search. A reference picture selection unit of theapparatus for the determination of motion vectors is configured toselect the first reference picture to which MV0 points and the secondreference picture to which MV1 points. Following the selection, thereference picture selection unit determines whether the first referencepicture or the second reference picture is used for the performance ofmotion vector refinement. In FIG. 3 , the first reference picture towhich motion vector MV0 points is selected for the search. Forperforming motion vector refinement, the search region 310 in the firstreference picture is defined around the candidate position to whichmotion vector MV0 points. The candidate search space positions withinthe search region 310 are analyzed to find a block most similar to atemplate block by performing template matching within the search spaceand determining a similarity metric such as the sum of absolutedifferences (SAD). As mentioned above, in one implementation, thetemplate is constructed based on a combination of samples from twodifferent reference pictures with the respective motion vector of MV0and MV1. The template matching is performed to find a block based on thepoint in search region 310 which is most similar to the template.Alternatively, in another implementation, the template is constructed tofind the similarity between a prediction block P0 generated based on MV0in L0 and a prediction block P1 generated based on MV1 in L1. To performthe refinement for MV0, the template matching comprises finding a blockbased on the point in search region 310 which is determined by asimilarity metric (such as SAD, for example) between P0 and P1. Thepositions of the search space 310 denote the positions on which the topleft corner of the template 320 is matched. As already mentioned above,the top left corner is a mere convention and any point of the searchspace such as the central point 330 can in general be used to denote thematching position.

The candidate position with the lowest SAD value is determined as themotion vector MV0″. In FIG. 3 , the position to which MV0″ points is ahalf-pel position and it differs from the initial estimate MV0 by 1.5pixel positions in vertical direction, while the horizontal directionremains the same.

According to the present disclosure, for bi-prediction ormulti-reference prediction, at least one motion vector of the currentblock is refined by calculation based on another refined motion vectorof the current block rather than by performing the template matching.FIG. 4 illustrates such refinement. In particular, the motion vectorMV1″ is calculated based on motion vector estimate MV1 and a function ofMV0″-MV0 instead of performing a second template matching. In theexample of FIG. 4 , the determination and refinement of the first motionvector MV0″ is performed as described above with reference to FIG. 3 .Moreover, motion vector MV1″ is calculated by subtracting from thesecond motion vector estimate MV1″ the difference between the MV0″ andMV0.

This approach exploits the fact that most of the motion in a videosequence falls in the category of “translational motion”. Intranslational motion, an object is moving at a constant speed (at leastbetween the frames that are close to each other in sampling time). Thismeans that an object is displaced in consecutive frames by the samepixel distance in x- and −y directions (if the temporal sampling periodis not varying over time). The invention utilizes the principle oftranslational motion to some extent.

In the above example a first motion vector MV0″ was refined by templatematching whereas a second motion vector was refined by calculation.However, according to the present disclosure, a selection process may befurther performed for setting, which motion vectors are to be refined bytemplate matching and which are to be determined by calculation. FIG. 5shows a block diagram of an example of a motion vector refiner 500.

The motion vector refiner 500 may be included within an apparatus fordetermining motion vectors 810 as illustrated in FIG. 8 . The apparatusmay be included in the inter-prediction unit 800 which may replace theinter-prediction unit 110 shown in FIG. 1 and/or in the inter-predictionunit 210 shown in FIG. 2 .

More specifically, an apparatus 810 for determining a first motionvector in a first reference picture and a second motion vector in asecond reference picture is provided. The first and second motionvectors can be applied in inter-prediction of a picture block in acurrent picture.

The apparatus 810 comprises a motion vector refiner 500 which, as shownin detail in FIG. 5 , further comprises a motion vector refinement unit530 configured to obtain an estimate MV0 of a first motion vector MV0″and determine the first motion vector MV0′″ by performing a searchwithin a search space specified based on the estimate MV0. The apparatusfurther comprises a motion vector calculation unit 550 configured toobtain an estimate MV1 of a second motion vector MV1″ and calculate thesecond motion vector MV1″ based on the estimate MV1 of the second motionvector MV1″ and based on the first motion vector MV0″.

In the example, the apparatus comprises a first branch which comprisesthe motion vector calculation unit 530 and the motion vector calculationunit 550, and a second branch which comprises the motion vectorcalculation unit 540 and the motion vector calculation unit 560, and aswitch 520 for enabling one of the two branches and disabling the otherone. The second branch is similar to the first branch and differs fromit mainly in that it outputs a first motion vector MV1″ and a secondmotion vector MV0″, by processing MV1 as an estimate of the first motionvector and MV0 as an estimate of the second motion vector.

More specifically, the motion vector refinement unit 540 is configuredto obtain an estimate MV1 of a first motion vector MV1″ and determinethe first motion vector MV1″ by performing a search within a searchspace specified based on the estimate MV1. The apparatus furthercomprises a motion vector calculation unit 560 configured to obtain anestimate MV0 of a second motion vector MV0″ and calculate the secondmotion vector MV0″ based on the estimate MV0 of the second motion vectorMV0″ and based on the first motion vector MV1″.

FIG. 5 shows application of the invention to bi-prediction in whichthere are two motion vectors in two respective reference picturesdetermined, namely motion vectors MV0″ and MV1″. Accordingly, alsoselection of the picture for refinement by template matching is alsoperformed simply by taking one of MV0 and MV1 for template matching andmaintaining the other one MV1 or MV0 respectively for calculation.

This process of decoder motion vector refinement (DMVR) is performed bythe apparatus 500. The motion vector refinement unit 530 or 540(depending on which of the respective motion vector estimates MV0 andMV1 the template search is to be performed) receives on an input 505 therespective motion vector estimate MV0 or MV1 from the motion vectorestimator 820 and sets a search region 310 around the MV0 or MV1. Thesize of the search region in FIGS. 3 and 4 is 3×3 integer pixels,half-pixel interpolated to 7×7, hence 49 positions in total. In general,the form and/or size of the search region may be different and thepresent invention works irrespectively of the size and shape of thesearch region. The search region may have a predetermined or apredefined size. In other words the shape and size of the search spacemay be fixed and specified in a standard. Alternatively, one of severalpossible forms and sizes may be selected manually by the user withinencoder settings and/or automatically based on the content of the video.

Some examples of sizes and shapes of a search space are shown in FIG. 6. The full triangle marks the central pixel of the search space, whilethe full circles mark the remaining search space positions. It is notedthat the search space may further be extended by fractional (half-pixel,quarter-pixel or the like) interpolation. The present disclosure isgenerally not limited to any particular pixel pattern.

For each of the positions or fractional positions within the searchregion, template matching using the template which approximates thecurrent block is performed, providing a SAD value for each search spacecoordinate. It is noted that integer pixel resolution and half-pixelresolution herein describes the resolution of the search space, i.e. thedisplacement of the searched positions to the non-refined motion vectorthat is input to the process. As a result the search coordinates do notnecessarily coincide with the actual pixel coordinates on the imageplane. In other words, integer pixel (sample) resolution of the searchspace does not necessarily mean that the search space positions arelocated on integer pixels of the reference image. Integer positions ofthe search space may coincide with fractional pixels of the referenceimage, especially if the initial motion vector MV0, MV1 pointed to afractional pixel within the reference image.

A comparison of the SAD values of the positions in the search region isperformed, determining the position with the minimum SAD value. Theposition with the minimum SAD value is determined as the motion vectorMV0″. As mentioned in the background section, SAD is only an example andany other metric such as MSE, SSIM, correlation coefficient or the likemay be used in general. The determined first motion vector MV0″ togetherwith an estimate of the second motion vector MV1 are passed to themotion vector calculation unit 550, where the determination (refinement)of the motion vector MV1″ is performed by calculation.

As a first exemplary approach, as already described above with referenceto FIG. 4 , the motion vector calculation unit 550 is configured tocalculate the second motion vector by adding to the estimate of thesecond motion vector a difference between the first motion vector andthe estimate of the first motion vector or a function of the differencebetween the first motion vector and the estimate of the first motionvector.

Adding the difference can be calculated as follows:MV1″=MV1+(MV0−MV0″)  (equation 1)

This equation 1 (with adding negative difference MV0″-MV0) works for thecase shown in FIG. 4 , when the two reference pictures are located oneof them before and the remaining other one of them after the currentpicture. As can be seen in FIG. 4 , in order to extrapolate regularmotion, the difference between the current motion vector and the firstmotion vector in a preceding picture is projected to the picturefollowing the current picture with the reversed sign.

If both reference frames for MV0 and MV1 are located either after orbefore the current picture, then the difference is directly addedwithout changing the sign resulting in:MV1″=MV1+(MV0″−MV0)  (Equation 2)

The location before/after mentioned above is meant to be in displayingorder. The displaying order of the pictures in a video may differ fromthe encoding/decoding order, i.e. from the order in which the codedpictures are embedded in the bitstream. The displaying order of thepictures may be given by Picture Order Count (POC). In particular, POCin H.265/HEVC is signaled in the slice header of each slice which is apicture or its portion.

The condition used to determine whether or not the two referencepictures are one following and one preceding the current picture may beimplemented using parameter POC as follows:(POCi−POC0)*(POCi−POC1)<0  (equation 3)wherein POCi is a display order (sequence number) of the currentpicture, POC0 is a display order of the first reference picture and POC1is a display order of the second reference picture. If the condition inequation 3 is true, than either the first reference picture is precedingand second reference picture is following the current picture or viceversa. If on the other hand the condition of equation 3 is not true,then either both reference pictures are preceding or both following thecurrent picture. However, please note that the condition may beimplemented in any way which enables to check whether or not the signsof the two difference expressions (POCi−POC0) and (POCi−POC1) are same.The above equation 3 is only an example taking multiplication “*” forits relatively simple implementation.

Adding the function of the difference can be calculated as follows incase of a bi-directional prediction in which one of the referencepictures precedes and the other one follows the current picture(condition in equation 3 being true):MV1″=MV1−f(MV0″−MV0)  (equation 4)

Again, if the two reference pictures were both preceding or bothfollowing the current picture in the display order (condition inequation 3 being false), then the calculation would add the differencewithout reversing the sign:MV1″=MV1+f(MV0″−MV0)  (equation 5)

For instance, the function may include scaling and/or clipping.

If the function is scaling, the calculation of the motion vector may beas follows:MV1″=MV1−k*(MV0″−MV0)=MV1+k*(MV0−MV0″)  (equation 6)wherein “*” stands for multiplication (scaling) and k is a scalingparameter. If k=1, equation 1 is obtained. With a fixed (predefined) k,equation 6 is applicable to cases where one of reference picturesprecedes and the other one follows the current picture. For the case inwhich both reference pictures are preceding or both are following thecurrent picture, the sign would need reversing:MV1″=MV1+k*(MV0″−MV0)=MV1−k*(MV0−MV0″)  (equation 7)

According to an embodiment, the value of the scaling parameter dependson a ratio between the respective distances of the first picture and thesecond picture to the current picture. Thus, the value of k is notpredefined and not fixed, but may vary depending on the referencepictures in which the motion vectors are defined. For instance, thescaling factor k may be given by:k=−(POCi−POC1)/(POCi−POC0)  (equation 8)

It is noted that the above equation 8 already takes into account thechange of the sign since the scaling parameter changes sign depending onwhether both reference pictures are on the same side (preceding orfollowing) of the current picture or on different sides of the currentpicture in the displaying order.

Even though it may be beneficial to take into account the distancebetween the respective first and second reference pictures to thecurrent picture, even if these distances are mutually different, settingk=1 as in equations 1 and 2 may still be applied. It is simpler and thequality of the refined motion vector may still be substantially higherthan not applying the refinement.

After the scaling operation, rounding or clipping operations can beapplied. Here, rounding refers to providing an integer or real numberwith a predefined accuracy closest to the input value. Clipping refersto removing digits with precision higher than a predefined accuracy fromthe input value. This may be particularly beneficial in view of afixed-point arithmetic applied in typical computation systems.

As the motion vector refinement is performed only on one of the tworeference pictures, the search space for performing template matching onis reduced by 50%.

After the above described processing for the current block in thecurrent picture, two reference pictures and their associated respectivemotion vectors MV0″ and MV1″ are provided at the output 580. Thesemotion vectors are used to determine the predictor for the currentblock, for instance, by averaging two respective predictors obtained bytaking respective portions of the reference pictures corresponding insize and shape to the current block and pointed to by the motion vectorsMV0″ and MV1″. This is performed by a prediction unit 890.

In general words, the prediction unit 890 determines the predictionblock by combining a portion of the first reference picture referred toby the first motion vector MV0″ and a portion of the second referencepicture referred to by the second motion vector MV1″.

The combined prediction signal can provide a better adaptation to theoriginal signal than uni-prediction, resulting in less residualinformation and possibly a more efficient compression.

As explained in the previous paragraph, the output motion vectors MV0″and MV1″ of the apparatus 500 may result in lower residual information.Hence, using them may improve the prediction performance compared to theinitial motion vectors MV0 and MV1. It is noted that the apparatus 500can be employed multiple times in order to improve the predictionperformance further. In this case the output of the first application ofapparatus 500 is considered as the input of the second application andequality between the motion vectors (equations 1 to 8) holds after eachapplication. In this case, since the motion vectors MV0″ and MV1″provide a better estimation of the current block after the firstapplication of apparatus 500, the template that is used in the MVrefiner unit 530 is constructed based on the samples pointed to by MV0″or samples pointed by MV1″ in the second application.

According to an exemplary embodiment, the apparatus 500 furthercomprises a reference picture selection unit 510 configured to obtaintwo reference pictures and select which of them is to be the firstreference picture and the second reference picture.

In other words, the reference picture selection unit 510 controls for acurrent block, which of its motion vectors is to be refined by templatematching and which is to be refined by calculation on the basis of arefinement of another refined motion vector. In the following, severalexamples are provided on how the selection process can be performed bythe reference picture selection unit 510.

FIG. 7 shows a schematic diagram illustrating a method 700 fordetermining a first motion vector in a first reference picture and asecond motion vector in a second reference picture according to anembodiment. For example, a digital video encoder 100 or decoder 200,described above, particularly the inter prediction unit 110 or 210, mayuse the process 700 to obtain the first and second motion vectors to beapplied in inter-prediction of a picture block in a current picture. Themethod 700 comprises a step 701 of obtaining an estimate of the firstmotion vector, a step 703 of determining the first motion vector byperforming a search within a search space specified based on theestimate of the first motion vector, a step 705 of obtaining an estimateof the second motion vector and a step 707 of calculating the secondmotion vector based on the estimate of the second motion vector andbased on the first motion vector.

Example 1

In a first example, the reference picture selection unit 510 decideswhich of the motion vector is to be determined by search in a searchspace on the basis of the list from which the reference picturebelonging to the respective motion vector was referred to. Inparticular, in order to signal predictor information which is used bythe decoder to recover the same predictor as used at the encoder withinthe bitstream, some codecs signal reference picture by including intothe bitstream an index associated in a list of reference pictures with aparticular reference picture. For instance the list of referencepictures (reference picture list) may be a table available at bothencoder and decoder and associating an index with a relative position ofa respective reference picture with respect to a current picture.

There may be more than one list of reference pictures. For instance,reference picture list L0 and reference picture L1 as usually used inH.265/HEVC. In order to signal in the bitstream the reference picture,at first a reference picture list is signaled followed by the indexwithin the signaled reference picture list.

The reference picture selection unit 510 is then advantageouslyconfigured to select either the first reference picture or the secondreference picture based on whether or not it is referred to in the bitstream by an index in a predefined list of reference pictures. In thiscontext the term “predefined” means fixedly defined for instance in astandard or defined for the entire video in a signaling. Exemplarily,the predefined list may be list L0. Then if the first reference picturewas referred to from reference picture list L0 whereas the second listwas referred to from reference picture list L1, the first motion vectorpointing to the first reference picture will be refined by templatematching since it was referred to from L0, whereas the second motionvector will be calculated since it was not referred to from L0. However,please note that the predefined list is not limited to the L0 list. Anyof the used reference picture lists may be predefined instead. Usually,L0 would list reference pictures in a closer neighborhood than L1.Alternatively, L0 may include more reference pictures preceding thecurrent picture in the displaying order, whereas L1 may include morepictures which follow the current picture in the displaying order.However, the present invention works irrespectively of which particulardifference there is between the two or more reference picture lists.

It is noted that a situation may occur in which both the first referencepicture and the second reference picture pointed to by the respectivefirst and second motion vectors are referred to from the same referencepicture list. For instance, both the first reference picture and thesecond reference picture may be referred to from picture the predefinedlist L0. Alternatively, if a current coding block applies bi-prediction,one reference picture from list L0 and one reference picture from listL1 must be indicated, where both reference pictures might be includedtogether in one (or both) of the reference lists. The reason is areference picture can be present in both lists (list L0 and list L1).

If the first reference picture and the second reference picture are bothincluded in the predefined list (such as L0), the picture with thehighest position in the list (L0) is selected as the reference pictureused for motion vector refinement by template matching of thecorresponding motion vector pointed thereto.

The same selection may be performed if both reference pictures areincluded in a list of reference pictures which is not predefined (suchas L1, if L0 is predefined). In other words, if the reference picturesare referred from a list of reference pictures other than the predefinedlist of reference pictures, the reference picture with the highestposition in the list is selected for template matching based motionvector refinement.

In summary, the reference picture selection unit 510 is configured to,if the two reference pictures are referred to in the bitstream by anindex in the same predefined list of reference pictures, select thefirst reference picture as the picture with the highest position in saidlist of reference pictures.

This example provides for a simple implementation without any additionalintelligent selection process. In particular, the reference pictureselection unit 510 controls a switch 520 and 570 to select the upper orthe lower branch in the block diagram of FIG. 5 when the parsedreference picture list is the predefined list (such as L0) and, if bothpictures are in the same list, also based on the index value of thereference picture.

Example 2

According to a second example, the reference picture selection unit MOis configured to select as the first reference picture (which is to berefined by template matching) a picture which has the lowest temporallayer among the two (or more) reference pictures for the current block.In other words, the reference picture selection unit 510 controls theswitches 520 and 580 to select the upper branch (530, 550) or the lowerbranch (540, 560) according to the temporal layer of the referencepictures to which motion vectors MV0 and MV1 are related.

It is noted that in FIG. 5 , the upper and lower branches do notnecessarily need to be implemented in a double manner. In general, inany of the embodiments and examples of the present disclosure, a singlemotion vector refinement unit 530 and a single motion vector calculationunit 550 are provided and merely the input to the single branch isswitched according to the control of the reference picture selectingunit 210.

In particular, the temporal layers of the two (or more) referencepictures are determined. The temporal layer of a reference pictureindicates the number of pictures, which have to be decoded before thereference picture can be decoded. The temporal layer is usually set atthe decoder which encodes the video motion picture in different temporallayers. It may be included into the bitstream. Thus, decoder may, basedon signaling information from the bitstream, determine which referencepictures belong to which temporal layer. Accordingly, then, the firstreference picture or the second reference picture with the lowertemporal layer is selected as the reference picture used for motionvector refinement by template matching. This approach may provide theadvantage of selecting for the template matching the reference picturewhich depends on less previously decoded pictures and thus, with lowerprobability of errors and artifacts. Accordingly, the motion vectorrefinement procedure is made more resilient.

Example 3

In a third example, the reference picture selection unit 510 isconfigured to select as the first reference picture (to be refined bytemplate matching) the picture which has the lowest base quantizationvalue. In other words, the reference picture selection unit 510 controlsthe switches 520 and 580 to select the upper branch (530, 550) or thelower branch (540, 560) according to the quantization parameter of thereference pictures to which motion vectors MV0 and MV1 are related.

Quantization value or quantization parameter in this context isinformation provided in the bitstream which enables to determinequantization step. In the well-known codecs such as H.264/AVC andH.265/HEVC, the quantization parameter enables to determine the value bywhich the coefficients to be quantized are to be divided. The larger thequantization value, the coarsest the quantization which typically leadsto a worse image quality after reconstruction. Thus, a lowerquantization value means that a higher quality of reconstructed imagemay be provided. Selection of reference picture with a lowerquantization parameter means that a reference picture with a betterquality is used for the motion vector refinement, which then also leadsto better refinement results.

The term “base” quantization value refers to a quantization value whichis common to a picture slice and used as a base for all blocks. Usually,such value is signaled for instance in the slice header. Then,typically, a difference from the base value is signaled on a block orprocessing unit basis.

However, the present invention is not limited to any particularsignaling or even existence of such value. The same effect can beachieved by determining the base quantization value for a pictureaccording to the quantization values of elements in the picture, forwhich the quantization value is signaled. In other words, the term basequantization value denotes a general quantization value for a picture.

Example 4

According to a fourth example, the reference picture selection unit 510is configured to select as the first reference picture the picture whichhas the lowest distance to the current picture. In other words, thereference picture selection unit 510 controls the switches 520 and 580to select the upper branch (530, 550) or the lower branch (540, 560)according to the distance of the reference pictures related to therespective motion vectors MV0 and MV1 to the current picture.

For instance, differences between the picture order count (POC) valuesPOC0 and POC1 of respective reference pictures related to the respectivemotion vectors MV0 and MV1 and the POC value POCi of the current pictureare determined. The POC value specifies the display order of thepictures rather than coding/decoding. Hence, a picture with POC=2 isdisplayed before a picture with POC=8. However, it is noted that as theinvention is not limited to application in the well known codecs such asH.264/AVC and H.265/HEVC, the difference between the reference pictureand the current picture may be determined in any other way, notdepending on the particular POC parameter.

The first reference picture related to the motion vector MV0 or thesecond reference picture related to the motion vector MV1 that has thesmaller absolute POC difference (between the reference picture and thecurrent picture) is selected as the reference picture to be used formotion vector refinement since it is expected that the motion vectors ofthe closer reference pictures are more precise and/or the referenceblock pointed to by the motion vectors is more similar to the currentblock. This may lead to a better quality of the refinement.

Example 5

According to a fifth example, the reference picture selection unit isconfigured to select the first reference picture and the secondreference picture such that the magnitude of the estimate of the firstvector is smaller than the magnitude of the estimate of the secondmotion vector. In other words, the reference picture selection unit 510controls the switches 520 and 580 to select the upper branch (530, 550)or the lower branch (540, 560) according to the length (magnitude) ofthe motion vectors MV0 and MV1 related to the respective referencepictures.

The absolute magnitudes of motion vectors MV0 and MV1 pointing to thefirst reference picture and the second reference picture are determinedusing standard procedures for the determination of an absolute value ofa vector. For instance, the squared values of each vector component of amotion vector are summed up. Either this sum or its square root may beused as the magnitude of the motion vector, but the calculation of asquare root has higher computational costs. Taking the motion vectorwith smaller magnitude for the refinement provides the advantage of ahigher likelihood to be determined correctly, assuming that motionbetween pictures is typically small.

Some information on an estimate of the first motion vector MV0, anestimate of the second motion vector MV1 and indices of the referencepictures to which MV0 and MV1 refer to may be received at the decoder asinput. The motion vector information is typically signaled on a blockbasis, the blocks can have different sizes. The same is true for thereference picture indication. A bitstream parser implemented as a partof the entropy decoding unit 203 obtains the motion vector informationfrom the bitstream. The motion information may be directly thecoordinates of the motion vector (coordinates with respect to the point(0, 0) given by the location of a block in the reference picture same asthe location of the current block in the current picture).Alternatively, a difference to a motion vector of a block precedingcurrent block in the decoding order may be signaled. This mayadvantageously be one of the spatial or temporal neighbors of thecurrent block.

According to another example, an apparatus for motion vectordetermination which also includes the motion vector refiner 500, furtherincludes a motion vector estimation unit 820 configured to determine theestimate of the first motion vector and the estimate of the secondmotion vector by selecting them from a set of motion vector candidatesbased on similarity of a template with a portion of the respectivepicture referred to by the motion vector candidate. In other words, themotion vector determination (of MV0 and MV1) is not necessarily based ontemplate matching in a search space defined in the reference picture.The search space may be given by a (merge) list which lists indexes inassociation with the respective motion vectors of the spatially ortemporally adjacent blocks or blocks in the proximity of the currentblock. This is to say that the present invention is not limited by theway in which the motion vector estimates MV0 and MV1 are derived beforethey are provided for refinement.

In summary, the bi-prediction operation of one coding block, the twoprediction blocks, from the motion vectors (MV) of list L0 and MV oflist L1 respectively, are combined to a single prediction signal, whichcan provide a better adaptation to the original signal thanuni-prediction, resulting in less residual information and possibly amore efficient compression. The bi-prediction decoding processing forthe current block in the current picture contains the followingprocessing steps.

First, an estimate of the first motion vector MV0 and an estimate of thesecond motion vector MV1 are received as input at the decoder side. Tworeference pictures, to which MV0 and MV1 point, are in the picturebuffer of the decoder as they have been already decoded beforeprocessing of the current picture. One of these reference pictures isselected for motion vector refinement by template matching, forillustration the reference picture to which MV0 points. For performingmotion vector refinement, a search region in the reference picture towhich MV0 points is defined around the candidate point to which theselected MV0 points. The candidate search space positions within thesearch region are analyzed by performing template matching with thecurrent block space and determining a similarity measure. The candidatesearch space position with the lowest dissimilarity value is determinedas the motion vector MV0″. The motion vector MV1″ is calculated based onMV1 and a function of MV0″-MV0 instead of performing a second templatematching.

According to an embodiment of the invention, the similarity comparisonis performed by comparing the samples pointed to by MV0″ and the samplespointed to by MV1″. According to FIG. 4 , any point pointed to by MV0″in the search space 310 has a counterpart motion vector MV1″ given byone or more of the equations 1 to 8. In order to compare similaritybetween sample blocks in the template matching process the sample blockpointed by MV0″ and sample block pointed by MV1″ can be used in afunction such as a SAD function. In this case the template consists onlyof samples that belong to the reference picture referred by MV1, andmotion vector refinement operation is performed on MV0. Moreover thetemplate is slightly changed for each point in the search space 310.

More specifically according to the embodiment the following steps areapplied:

Step 1: The similarity between the samples pointed to by input motionvector MV0 and the samples pointed to by input motion vector MV1 iscomputed. The input motion vector MV0 points to the point 330 in thesearch space.

Step 2: A second point in the search space 310 is selected, which isdifferent from the point 330. The second point is denoted by MV0″.

Step 3: The motion vector MV1″ is computed using one or more of theequations 1 to 8, based on MV0″, MV0 and MV1.

Step 4: The similarity between the samples pointed to by input motionvector MV0″ and the samples pointed to by input motion vector MV1″ iscomputed. If the similarity is higher than the computed value in Step 1,the MV0″ and MV1″ pair is selected as the refined motion vectors.Otherwise the MV0 and MV1 pair (i.e. the initial motion vector pair) isselected as refined motion vectors.

Steps 2, 3 and 4 can be repeated to evaluate more candidate points inthe search space 310. If no search points are left in the search space,then the refined motion vectors are output as the final refined motionvectors.

In step 4, it is clear that if the similarity metric (such as SAD, SSIM,etc.) might result in the highest similarity at the initial point 330that is pointed by initial motion vector MV0 whose counterpart isinitial motion vector MV1 according to one or more of the equations 1 to8. If this is the case the refined motion vectors hence the output ofthe motion vector refinement process is considered to be MV0 and MV1.

Any motion vector pair MV0″ and MV1″ which is the output of theapparatus 500 must obey the rules described in one or more of theequations 1-8. The specific details of the construction of the templateand the similarity metric that is used in the template matchingoperation might change without affecting the invention and its benefit,which is the reduction of search points to be checked by pairing of twomotion vectors.

According to an embodiment of the present invention, apart from thebi-prediction processing performed in the inter-prediction unit asdescribed above, the other processing steps for encoding and decodingfollow the standard H.265/HEVC.

However, in general, the present invention is applicable to any videodecoder for decoding a plurality of pictures from a bitstream. Suchdecoder may then comprise an inter-prediction unit including theapparatus according to any of claims 1 to 11 and a prediction unit fordetermining a prediction block according to a portion of the firstreference picture referred to by first motion vector and a portion ofthe second reference picture referred to by the second motion vector.The decoder may further comprise a bitstream parser. The bitstreamparser may be implemented, for example, as a part of the entropydecoding unit 203 and configured to obtain from the bitstream theestimate of the first motion vector and the estimate of the secondmotion vector. The video decoder may further comprise a reconstructionunit 211 configured to reconstruct the current block according to theprediction block.

On the other hand, a video encoder for encoding a plurality of picturesinto a bitstream may comprise an inter-prediction unit including theapparatus according to any of claims 1 to 12 and a prediction unit fordetermining prediction block according to a portion of the firstreference picture referred to by first motion vector and a portion ofthe second reference picture referred to by the second motion vector, abitstream former which is implemented as a part of the entropy encodingunit 103 and configured to include into the bitstream the estimation ofthe first motion vector and the estimation of the second motion vector,and a reconstruction unit 111 configured to reconstruct the currentblock according to the prediction block and storing the reconstructedblock in a memory.

The inter-prediction decoding processing described above is not limitedto the use of two reference pictures. Alternatively, more than tworeference pictures and the associated motion vectors may be considered.In this case, reference picture selection unit selects more than tworeference pictures out of which one reference picture is used for motionvector refinement. The selection of the reference picture which is usedfor motion vector refinement is using one of the approaches described inexamples 1 to 5 discussed above. The remaining motion vectors areadjusted using the estimate of the respective motion vector and themotion vector resulting from motion vector refinement. In other words,the present invention as described above may also work, if amulti-reference prediction is performed. For instance, if there arethree reference pictures and three respective motion vectors, one of thethree motion vectors may be determined by refinement by templatematching and two others calculated. This provides complexity reduction.Alternatively, two of the motion vectors are determined by refinement bytemplate matching and one is calculated based on one or both of therefined motion vectors. As is clear to those skilled in the art, thepresent invention is extendable to any number of reference pictures andthe corresponding motion vectors used to construct predictor for thecurrent block.

The invention has the effect to make it possible to performbi-prediction at a decoder with reduced processing load and memoryrequirements. It can be applied at any decoder, may it be included in acoding device and/or in a decoding device, i.e. at the encoder sider orthe decoder side.

The motion vector refinement as described above can be implemented as apart of encoding and/or decoding of a video signal (motion picture).However, the motion vector refinement may also be used for otherpurposes in image processing such as movement detection, movementanalysis, or the like.

The motion vector refinement may be implemented as an apparatus. Suchapparatus may be a combination of software and hardware. For example,the motion vector refinement may be performed by a chip such as ageneral purpose processor, or a digital signal processor (DSP), or afield programmable gate array (FPGA), or the like. However, the presentinvention is not limited to implementation on a programmable hardware.It may be implemented on an application-specific integrated circuit(ASIC) or by a combination of the above mentioned hardware components.

The motion vector refinement may also be implemented by programinstructions stored on a computer readable medium. The program, whenexecuted, causes the computer to perform the steps of obtainingestimates of the motion vectors, determining the first reference pictureand the second reference picture based on the estimates, performingmotion vector refinement of the first motion vector, calculating thesecond motion vector based on the estimates of the motion vectors andthe first motion vector after refinement. The computer readable mediumcan be any medium on which the program is stored such as a DVD, CD, USB(flash) drive, hard disc, server storage available via a network, etc.

The encoder and/or decoder may be implemented in various devicesincluding a TV set, set top box, PC, tablet, smartphone, or the like. Itmay be a software, app implementing the method steps.

What is claimed is:
 1. An apparatus for performing inter-prediction fora picture block using a first motion vector associated with a firstreference picture of a video and a second motion vector associated witha second reference picture of the video, the apparatus comprising: aprocessor; and at least one memory comprising a non-transitorycomputer-readable medium including computer-executable instructionsthat, when executed by the processor, cause the apparatus to carry out amethod comprising: obtaining an estimate of the first motion vector;determining, by performing a search within a search space specifiedbased on the estimate of the first motion vector, the first motionvector; obtaining an estimate of the second motion vector; calculatingthe second motion vector by adding a negative difference between thefirst motion vector and the estimate of the first motion vector to theestimate of the second motion vector; and applying the first motionvector and the second motion vector in association with aninter-prediction operation performed on the picture block.
 2. Theapparatus according to claim 1, wherein the calculating the secondmotion vector comprises adding to the estimate of the second motionvector a function of the difference between the first motion vector andthe estimate of the first motion vector.
 3. The apparatus according toclaim 2, wherein the function includes scaling by a scaling factorand/or clipping.
 4. The apparatus according to claim 3, wherein thevalue of the scaling factor depends on a ratio between the respectivedistances of the first reference picture and the second referencepicture to the current picture.
 5. The apparatus according to claim 1,wherein the method further comprises: obtaining two reference pictures;selecting the first reference picture from among the two referencepictures; and selecting the second reference picture from among the tworeference pictures.
 6. The apparatus according to claim 5, wherein themethod comprises: selecting the first picture or the second picture inaccordance with the respective first picture or second picture beingreferred to, in a bit stream that also includes the coded picture blockof the video, by an index relating to a predefined list of referencepictures among at least two lists of reference pictures, wherein a listof reference pictures associates an index with the position of areference picture with respect to the current picture.
 7. The apparatusaccording to claim 6, wherein the method comprises: selecting as thefirst reference picture, in accordance with the two reference picturesbeing referred to in the bitstream by an index in the same predefinedlist of reference pictures, the picture with the highest position insaid list of reference pictures.
 8. The apparatus according to claim 5,wherein the method comprises: selecting, as the first reference picture,the picture that has a lowest temporal layer among the two pictures,wherein a temporal layer of a reference picture indicates a number ofpictures that must be decoded to decode the reference picture.
 9. Theapparatus according to claim 5, wherein the method comprises: selecting,as the first reference picture, the picture that has a lowest basequantization value, wherein a base quantization value refers to aquantization value that is common to a picture slice and used for allblocks.
 10. The apparatus according to claim 5, wherein the methodcomprises: selecting, as the first reference picture, the picture thathas a lowest distance to the current picture.
 11. The apparatusaccording to claim 5, wherein the method comprises: selecting the firstreference picture and the second reference picture such that theestimate of the first motion vector is smaller in magnitude than theestimate of the second motion vector.
 12. The apparatus according toclaim 1, wherein the method further comprises: determining the estimateof the first motion vector and the estimate of the second motion vectorby selecting the estimate of the first motion vector and the estimate ofthe second motion vector from a set of motion vector candidates based onsimilarity of a template with a portion of the picture referred to bythe respective motion vector candidate.
 13. The apparatus of claim 1,configured to carry out functionality of a video decoder that decodes aplurality of pictures from a bitstream, the apparatus further comprisingcomputer-executable instructions that, when executed by the processor,cause the apparatus to further perform: determining a prediction blockaccording to a portion of the first reference picture referred to by thefirst motion vector and a portion of the second reference picturereferred to by the second motion vector; obtaining, from the bitstream,the estimation of the first motion vector and the estimate of the secondmotion vector; and reconstructing the current block according to theprediction block.
 14. The apparatus of claim 1, configured to carry outfunctionality of a video encoder that encodes a plurality of picturesinto a bitstream, the apparatus further comprising computer-executableinstructions that, when executed by the processor, cause the apparatusto further perform: determining a prediction block according to aportion of the first reference picture referred to by the first motionvector and a portion of the second reference picture referred to by thesecond motion vector; including into the bitstream the estimate of thefirst motion vector and the estimate of the second motion vector;reconstructing the current block according to the prediction block; andstoring the reconstructed block in the at least one memory.
 15. A methodfor performing inter-prediction for a picture block using a first motionvector associated with a first reference picture of a video and a secondmotion vector associated with a second reference picture of the video,the method comprising: obtaining an estimate of the first motion vector;determining, by performing a search within a search space specifiedbased on the estimate of the first motion vector, the first motionvector; obtaining an estimate of the second motion vector; calculatingthe second motion vector by adding a negative difference between thefirst motion vector and the estimate of the first motion vector to theestimate of the second motion vector; and applying the first motionvector and the second motion vector in association with aninter-prediction operation performed on the picture block.
 16. Anon-transitory computer-readable medium carrying executable instructionsthat, when executed by a processor in an apparatus, cause the apparatusto perform a method for performing inter-prediction for a picture blockusing a first motion vector associated with a first reference picture ofa video and a second motion vector associated with a second referencepicture of the video, the method comprising: obtaining an estimate ofthe first motion vector; determining, by performing a search within asearch space specified based on the estimate of the first motion vector,the first motion vector; obtaining an estimate of the second motionvector; calculating the second motion vector by adding a negativedifference between the first motion vector and the estimate of the firstmotion vector to the estimate of the second motion vector; and applyingthe first motion vector and the second motion vector in association withan inter-prediction operation performed on the picture block.
 17. Themethod of claim 15, wherein the recited operations are carried out by avideo encoder.
 18. The method of claim 15, wherein the recitedoperations are carried out by a video decoder.
 19. The apparatus ofclaim 1, wherein the recited operations are carried out by a videoencoder.
 20. The apparatus of claim 1, wherein the recited operationsare carried out by a video decoder.